{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BIDMach: parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll explore automated parameter exploration by grid search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import $exec.^.lib.bidmach_notebook_init\n",
    "if (Mat.hasCUDA > 0) GPUmem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: Reuters RCV1 V2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is the widely used Reuters news article dataset RCV1 V2. This dataset and several others are loaded by running the script <code>getdata.sh</code> from the BIDMach/scripts directory. The data include both train and test subsets, and train and test labels (cats). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var dir = \"../data/rcv1/\"             // adjust to point to the BIDMach/data/rcv1 directory\n",
    "tic\n",
    "val train = loadSMat(dir+\"docs.smat.lz4\")\n",
    "val cats = loadFMat(dir+\"cats.fmat.lz4\")\n",
    "val test = loadSMat(dir+\"testdocs.smat.lz4\")\n",
    "val tcats = loadFMat(dir+\"testcats.fmat.lz4\")\n",
    "toc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets enumerate some parameter combinations for learning rate and time exponent of the optimizer (texp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val lrates = col(0.03f, 0.1f, 0.3f, 1f)        // 4 values\n",
    "val texps = col(0.3f, 0.4f, 0.5f, 0.6f, 0.7f)  // 5 values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to enumerate all pairs of parameters. We can do this using the kron operator for now, this will eventually be a custom function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val lrateparams = ones(texps.nrows, 1) ⊗ lrates\n",
    "val texpparams = texps ⊗ ones(lrates.nrows,1)\n",
    "lrateparams \\ texpparams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the learner again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val (mm, opts) = GLM.learner(train, cats, GLM.logistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep things simple, we'll focus on just one category and train many models for it. The \"targmap\" option specifies a mapping from the actual base categories to the model categories. We'll map from category six to all our models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val nparams = lrateparams.length\n",
    "val targmap = zeros(nparams, 103)\n",
    "targmap(?,6) = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opts.targmap = targmap\n",
    "opts.lrate = lrateparams\n",
    "opts.texp = texpparams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mm.train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val (pp, popts) = GLM.predictor(mm.model, test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And invoke the predict method on the predictor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.predict\n",
    "val preds = FMat(pp.preds(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.model.asInstanceOf[GLM].mats.length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although ll values are printed above, they are not meaningful (there is no target to compare the prediction with). \n",
    "\n",
    "We can now compare the accuracy of predictions (preds matrix) with ground truth (the tcats matrix). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val vcats = targmap * tcats                                          // create some virtual cats\n",
    "val lls = mean(ln(1e-7f + vcats ∘ preds + (1-vcats) ∘ (1-preds)),2)  // actual logistic likelihood\n",
    "mean(lls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more thorough measure is ROC area:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val rocs = roc2(preds, vcats, 1-vcats, 100)   // Compute ROC curves for all categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(rocs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val aucs = mean(rocs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maxi2 function will find the max value and its index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val (bestv, besti) = maxi2(aucs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And using the best index we can find the optimal parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "texpparams(besti) \\ lrateparams(besti)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Write the optimal values in the cell below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Note:</b> although our parameters lay in a square grid, we could have enumerated any sequence of pairs, and we could have searched over more parameters. The learner infrastructure supports more intelligent model optimization (e.g. Bayesian methods). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala211",
   "nbconvert_exporter": "script",
   "pygments_lexer": "scala",
   "version": "2.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
